fix(healthcheck): prevent ClickHouse broken-pipe errors in the telemetrystore healthcheck - #194
Open
julpar wants to merge 1 commit into
Open
fix(healthcheck): prevent ClickHouse broken-pipe errors in the telemetrystore healthcheck#194julpar wants to merge 1 commit into
julpar wants to merge 1 commit into
Conversation
…trystore healthcheck
The ClickHouse healthcheck uses `wget --spider`. The `--spider` option closes
the connection before it reads the response body. ClickHouse then fails to
write the `/ping` response. It records the failure as an `Error` with a full
stack trace.
The healthcheck runs every 30 seconds. Each failure writes two error lines.
This adds approximately 5700 stack traces each day. The container stays
healthy, because the check still exits 0, so the noise is not obvious. The
noise hides real ClickHouse errors.
This change replaces `--spider` with `-O-`, which reads the response body.
Measured on clickhouse-server 25.5.6, with 25 requests for each command:
wget --spider -q 0.0.0.0:8123/ping 22/25 errors
wget -q -O- 0.0.0.0:8123/ping 0/25 errors
The change applies to the coolify and ecsterraform castings. These castings
use the `0.0.0.0` host. The dockercompose and dockerswarm castings use
`http://localhost:8123/ping` and stay unchanged.
Do not change the host to `localhost` to solve this problem. In the ClickHouse
image, `localhost` resolves to `::1` first. If the host disables IPv6, the
connection is refused and the healthcheck fails.
Refs SigNoz/signoz#5140
This was referenced Sep 5, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes
Problem
The ClickHouse healthcheck uses
wget --spider. The--spideroption closes the connection before it reads the response body. ClickHouse then fails to write the/pingresponse, and records the failure as anErrorwith a full C++ stack trace.The healthcheck runs every 30 seconds, and each failure writes two error lines. This adds about 5,700 stack traces a day. The container stays
healthythe whole time, because the check still exits 0, so nothing surfaces the problem. The noise hides real ClickHouse errors.This is the cause of the long-standing report in SigNoz/signoz#5140 (open since June 2024). A user identified the healthcheck there in September 2025, and a maintainer asked for a PR. This is that PR.
Fix
Replace
--spiderwith-O-, sowgetreads the response body.Measurements
On
clickhouse-server:25.5.6, running each command 25 times and counting newStaticRequestHandlererror lines:wget --spider -q 0.0.0.0:8123/pingwget -q -O- 0.0.0.0:8123/pingVerified on a live single-node deployment: the error count went from a steady stream to zero, and the container stays
healthywithFailingStreak=0.Scope
Only the
coolifyandecsterraformcastings are affected, because they use the0.0.0.0host. Thedockercomposeanddockerswarmcastings usehttp://localhost:8123/pingand are left unchanged.One thing worth flagging
Please do not fix this by changing the host to
localhost. In the ClickHouse image,/etc/hostsresolveslocalhostto::1first. On a host with IPv6 disabled, the connection is refused:That variant produces no broken-pipe errors only because the request never reaches ClickHouse, and it would make the healthcheck fail. It may be worth a separate look at whether the
dockercomposeanddockerswarmcastings are exposed to this on IPv6-disabled hosts. I have not tested those flavors, so I left them alone.Refs SigNoz/signoz#5140